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Part I. Introduction 



Last year many of America’s political, business, and education leaders convened at a 
National Education Summit in Washington, D.C., to discuss ways to improve the nation’s 
high schools. Following the summit, 22 states joined the American Diploma Project 
Network, an initiative to better align high school graduation requirements with the skills 
young adults need to do well in today’s labor market and higher education. 

High school graduates stand to benefit enormously from such policies. But increasing the 
value of the high school diploma will do little to help those who don’t earn one. 

Therefore, Achieve and Jobs for the Future (JFF) have joined other organizations and 
foundations in calling for “dual goals” to guide high school improvement efforts. 
Policymakers must find ways to raise graduation rates even as they simultaneously work 
to raise standards for graduation. 

Unfortunately, most dropout prevention programs conducted over the previous 30 years 
have garnered only disappointing results.^ And many people still consider the two goals 
to be in natural conflict with one another — believing that the cost of raising standards 
must inevitably be higher dropout rates, or that raising graduation rates can only be 
accomplished by lowering standards. 

Recent experience suggests that worries about such zero-sum trade-offs might be 
misguided. During the last decade, a number of states and districts have raised graduation 
standards without seeing the plummeting graduation rates that critics of such policies 
often prophesize. But holding graduation rates steady cannot be the policy goal. 

Thirty years ago, most teenagers who dropped out of high school could expect to find a 
well-paying job, and most who worked hard could expect to climb the economic ladder. 
But the world has changed. Today, high school dropouts face diminishing opportunities 
and a lifetime of financial struggle. In fact, the median earnings of families headed by a 
high school dropout declined by nearly a third between 1974 and 2004.^ 

And wage declines are only part of the picture. Fast year, Columbia University hosted a 
conference on the cost of educational failure that presented evidence that high school 
dropouts are more likely to be unemployed, receive public assistance, commit crimes and 
become incarcerated. At the same time, they are less likely to receive job-based health 
insurance and pension plans, be healthy and live as long, and vote and make other kinds 
of civic contributions.^ 

That means the larger society stands to benefit from dropout prevention as well. Keeping 
all students in high school and graduating more young people with better skills will save 
millions of taxpayer dollars, greatly expand tax revenues, reduce crime, and improve 
citizenship. 
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Fortunately, recent research and experience convey three important lessons: 

1) The dropout problem is not an inevitable, immutable feature of American 
education. Demographics matter, but what happens in schools has a great impact 
on whether students stay in school and graduate. Recent research suggests that, 
even for students who have difficult home lives, dropping out has much to do 
with how schools operate and the educational experiences students have within 
them. Moreover, the conventional wisdom that dropping out is a highly 
idiosyncratic process driven by entirely personal factors is not true for most 
students who leave school. Most dropouts follow identifiable pathways through 
the education pipeline. 

2) We can do a much better job predicting which students are most likely to drop 
out. An ongoing study in Philadelphia can now identify half of all eventual high 
school dropouts as early as 6th grade."^ And a group of Chicago researchers can 
predict 85 percent of eventual dropouts in that city’s public school system based 
on just a few facts about the system’s 9th graders.® 

3) Finally, we know more than ever before about how schools contribute to high 
dropout rates and what educators can do to solve the problem. (See the companion 
JFF paper. Improving Outcomes for Struggling and Out-of- School Youth). 

This white paper was prepared for Staying the Course: High Standards and Improved 
Graduation Rates, a joint project of Achieve and JFF funded by Carnegie Corp. of New 
York. Its goal is to provide policymakers with an overview of research about the dropout 
problem and the best strategies for building an early warning data system that can signal 
which students and schools are most in need of interventions. 

As pressure mounts to do something about the dropout problem, many school systems 
will be tempted to skip questions about how to predict which students are most at risk of 
dropping out and simply begin with reforms meant to solve the problem. Leaders might 
assume that educators can do a pretty good job guessing which students are at risk based 
on subjective judgments. Or they might simply decide not to spend money on data 
systems but rather invest all of their dollars in interventions and reforms instead. As this 
paper attempts to demonstrate, however, such decisions can have a variety of negative 
consequences. 

Indeed, the cost of building an accurate Early Warning System is relatively small 
compared with the cost of providing programmatic interventions or systemwide reforms 
meant to increase graduation rates. But the payoff of basing interventions on accurate 
data can be huge. A large school system that invests in better data to support dropout 
prevention can obtain much better results for hundreds of thousands or even millions of 
dollars less than a similar system whose leaders decide to skip that step. 
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Part II. What Kinds of Students Drop Out of High School, 
AND Why? 



Knowing how best to predict dropping out requires knowing something about the kinds 
of students who drop out of high school and the reasons they give for doing so. 
Researchers have identified three kinds of factors that put students at greater risk for 
dropping out: 

1. Students’ Social Background 

Not surprisingly, a voluminous body of research has shown that factors related to 
students’ social and family background can increase or decrease the risk of dropping out: 

• Students who are poor, who are members of minority groups, who are male, 
who transferred among multiple elementary and middle schools, and who are 
overage for their grade are more likely to drop out of high school. 

• Students who come from single parent families, have a mother who dropped 
out of high school, have parents who provide low support for learning, or have 
parents who do not know their friends’ parents well also are placed at greater 
risk. 

• Finally, studies have suggested that teenagers who take on adult 
responsibilities — becoming a parent, getting married, and holding down a 
job — are also more likely to leave school without a diploma.^ 

However, several studies also have found that some social background indicators are 
neither good nor bad, per se, but instead influence dropping out in very complex ways. 
For example, one study found that mobility between schools before 8th grade increases 
the risk of dropping out, but mobility during early high school can be beneficial.^ 

Another found that holding a job outside of school can exert a negative, neutral, or even 
beneficial impact depending on a student’s gender and type of job he or she has.® 

2. Students’ Educational Experiences 

For a long time, researchers investigating the causes of the dropout problem focused 
mainly on the social and demographic characteristics of dropouts. Partly as a result, many 
educators still believe that dropping out is caused exclusively by students’ personal and 
family characteristics — things that don’t have anything to do with education and that 
educators can’t control. As one school principal told a writer for the American School 
Board Journal, “If you want to know why some students drop out, look at their parents 
— they pass their low aspirations on to their kids.”^ 



Achieve, Inc. 



June 2006 



Page 4 






As often turns out to be the ease, however, the eonventional wisdom is wrong. A large 
body of more reeent researeh has doeumented that while demographie faetors are 
important, they don’t tell the whole story or even the most salient part of it. Students’ 
edueational experienees play a large role in shaping the dropout problem, too. 

This is refleeted quite elearly in surveys of reeent dropouts. For example, in 1990 a 
federal researeh projeet surveyed a representative sample of dropouts about why they had 
left sehool. Out of 2 1 possible reasons refleeting a wide range of edueational and 
personal faetors, 5 1 pereent of dropouts reported, “I didn’t like sehool,” and 44 pereent 
said, “I was failing.” A team of researehers who further analyzed the results found that 
“in eaeh of the raeial/ethnie as well as gender groups, sehool-related faetors are the most 
eited reasons for dropping out.”^° 

A survey of the nation’s high sehool students eondueted in 2002 obtained similar results. 
When students who had ever eonsidered dropping out of sehool were asked why, 76 
pereent said sehool was boring and 42 pereent said they were not learning enough — 
responses that onee again beat out a long list of other possibilities by a substantial 
margin.^ ^ 

More sophistieated studies have helped verify and elarify the edueation-related faetors 
that play a role in fueling dropout rates. Researehers have identified two important 
eategories of edueational risk faetors: 

1) Aeademie performanee : Students who struggle in the elassroom and fall behind 
aeademieally are more likely to drop out. Low grades, low test seores, Fs in 
English and math, falling behind in eourse eredits, and being held baek one or 
more times all have been linked to lower ehanees for graduation. 

2) Edueational engagement : Students who beeome disengaged from sehool and 
develop diseiplinary problems are more likely to drop out. High rates of 
absenteeism or truaney, poor elassroom behavior, less partieipation in 
extraeurrieular aetivities, and bad relationships with teaehers and peers all have 
been linked to lower ehanees for graduation. 

Edueation-related faetors should be important to polieymakers and edueators eoneerned 
with redueing dropout rates both beeause they are practical and beeause they are 
predictive. First, sueh faetors deseribe something that takes plaee inside of sehool rather 
than something that happens in the home or that students bring with them from outside. 
As sueh, they ean be helpful in determining the kinds of interventions that students might 
need and that edueation systems ean provide. 

Seeond, edueational experienees ean be powerful predietors of dropping out, either along 
with demographie faetors or, in some kinds of eommunities, even apart from them. 
Espeeially in urban distriets, where high proportions of students have demographie risk 
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factors, very low academic performance and educational engagement can help 
discriminate the ones most likely to be on the path to dropping out. 

The same is true in other more homogeneous settings. For example, one study conducted 
among white, French-speaking Canadians found that, “once school experience is known, 
family, behavioral, personal, and social experiences do very little to improve the 
predictive accuracy.”^ ^ 

3. School Characteristics 

Beginning in the late 1980s, some researchers became concerned that their peers were 
focusing entirely on individual-level factors behind dropping out, wondering whether 
schools themselves could be contributing to the problem, too.^^ Since then, a convincing 
and ever-growing body of research has validated their suspicions: School-level factors 
play a significant role in determining whether students will earn a diploma. Institutions 
matter as much as individuals, and attending a high school with certain characteristics can 
itself be a risk factor for dropping out. 

That’s not to say that some schools literally kick students out (although that does 
happen), but rather that variations in the dropout rate across schools in a district or a state 
are not just a function of the students they teach. For example, the Consortium on 
Chicago School Research has documented that dropout rates vary widely across Chicago 
high schools — even after researchers adjust for a host of individual risk factors, 
including race, gender, prior academic achievement, family socioeconomic status and 
whether students are overage when they enter 9th grade. 

What characteristics of high schools make them more or less good at “holding” teenagers 
until graduation? Several recent studies have found that, student risk factors being equal, 
high schools with smaller enrollments, better interpersonal relationships among students 
and adults, teachers who are more supportive of students, and a curriculum that is both 
more focused and more rigorous exhibit lower dropout rates. And the benefits are 
especially great for low-achieving, low-income students. 

School effects can be quite strong. For example, researchers Valerie Lee and Robert 
Croninger found that high schools with highly supportive teachers cut the probability of 
dropping out in half Lee and David Burkam found that schools that offered fewer math 
courses below the level of Algebra I reduced the odds of dropping out by 28 percent, and 

1 7 

those that offered Calculus reduced the odds by 55 percent. 
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Part TTT. Can We Predict Which Students Wht. Drop Out? 

What Doesn’t Work? 

Knowing the research described in the last section is important, but it does not 
automatically lead to a practical, reliable system for predicting whether any particular 
student will drop out. Consider the lesson of past dropout prevention efforts, which 
experienced a surge in popularity during the 1980s and early 1990s. 

Following publication of A Nation at Risk, a number of major counter-reports lamented 
lack of attention to graduation rates on the part of the burgeoning “educational 
excellence” movement. Foundation officials and others worried that raising graduation 
standards would help the college-bound, but leave other high school students stranded in 
the “rising tide of mediocrity.” ® 

States and districts responded. “By the late 1980s, virtually every major school system 
had grappled with the issue of high school dropouts — on the one hand, by trying to 
identify the extent of the problem in their communities, and, on the other hand, by 
committing resources to address the problem,” researcher Melissa Roderick observed in 
1993 19 federal government pitched in by contributing $214 million between 1988 
and 1994 for a School Dropout Demonstration Assistance Program.^° 

As programs proliferated, experts become concerned about the methods being used to 
identify potential dropouts for interventions. For example, a brief published in 1989 by 
the National Dropout Prevention Center at Clemson University reported that schools 
often used “checklists” to assess students’ risk of dropping out. Such instruments 
typically were developed by examining the research literature to find out what 
characteristics were associated with dropouts or by interviewing dropouts themselves. 
The authors warned that the checklist approach had several major weaknesses: 

First, it is difficult to know how many characteristics must be checked before 
deciding if a student is at risk. Second, this gross approach to identification often 
leads to students being misclassified at risk and placed in prevention programs 
although they probably would not have dropped out had nothing been done for 
them. Third, educators frequently borrow instruments designed for use with 
students in program locales very different from their own, limiting generalization 
of characteristics on the checklist.^^ 

A study conducted by Phillip Gleason and Mark Dynarski of Mathematica Policy 
Research, Inc., suggests that such concerns were quite valid. They examined the 
effectiveness of widely used risk factors by examining data from a sample of program 
sites funded by the federal School Dropout Assistance Program. No single risk factor 
yielded an actual dropout rate of more than 28 percent. (That is, the most powerful risk 
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factor was not very predictive: More than 70 percent of students with that risk factor 
would have graduated anyway.) 

Some experts contend that risk factors are cumulative in nature, such that the greater 
number of risk factors a student has, the higher the probability he or she will drop out. 
Therefore, Gleason and Dynarski examined whether prevention programs using a 
combination of risk factors would better identify students who truly required 
interventions. Again, however, no combination of risk factors yielded a dropout rate of 
more than 28 percent. 

The Mathematica researchers concluded that “dropout prevention programs often serve 
students who would not have dropped out, and [they] do not serve students who would 
have dropped out.”^^ In other words, knowing that an indicator has been “correlated” 
with dropping out might not automatically make it a great, or even very good, predictor 
of whether a particular student will graduate or not. 

To understand why, consider a hypothetical example: If a district survey revealed that 40 
percent of dropouts had average grades below C, compared with only 20 percent of 
graduates, and, further, that 35 percent of dropouts had been suspended, compared with 
only 7 percent of graduates, one could say that dropouts in the district “were twice as 
likely to have had low grades and five times as likely to have had behavior problems.” 
That certainly makes it seem like the district could use those risk factors to “predict” 
dropping out. 

However, those same numbers would also mean that the majority of the district’s 
dropouts (60 percent) had grades of C or above, and approximately two-thirds of them 
had never been suspended. Prediction requires more than simply knowing which personal 
and educational characteristics dropouts are more likely to have.^^ 

To that end, the 1989 Dropout Prevention Center brief encouraged educators to adopt an 
alternative to checklists. The authors described a process whereby programs could 
administer a questionnaire to two groups — dropouts and graduates — and conduct a 
statistical analysis that considered all of their characteristics at once (often called a 
“regression analysis”) to develop a “statistically generated prediction formula.”^'^ 

Gleason and Dynarski reasoned that some programs might be using such a method, so 
they tested whether that approach would have helped the federally funded dropout 
prevention programs better identify students who were in real danger of dropping out of 
school. The good news was that it did. The bad news was, not by enough. Once again, the 
Mathematica researchers found that even if dropout prevention programs employed 
statistical formulas, they were apt to “serve more students who do not need dropout 

25 

prevention services than students who do need them.” 
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What W ould W ork Better? 



The answer lies in how Gleason and Dynarski tested the predictive power of indicators 
commonly used by prevention programs. Simply put, “longitudinal data allowed risk 
factors to be assessed.”^^ 

Longitudinal data, information collected and accumulated over time, permit analysts to 
follow the progress of individual students as they progress from grade to grade as 
members of a “cohort,” a group of students who start out in the same grade at the same 
time. Such data make it possible to observe what happens to students who develop risk 
factors at any point along the way, and thereby to paint a more detailed, nuanced portrait 
of the patterns and pathways students tend to follow as they move toward dropping out or 
graduating from high school. 

Of course, research using longitudinal information dates back to before the 1990s. But 
many researchers studying dropouts have been confined to using national data sets that 
wash out important differences among different regions and communities. Some 
researchers now believe that the power of place can be as important as the power of time 
for unmasking the pathways students take to dropping out.^^ 

Since the early 1990s, several longitudinal studies zeroing in on individual school 
districts have shed considerable new light on how, in addition to why, students drop out 
of school. Below we describe each study in some detail, because, taken together, they 
provide a rough blueprint for how districts can better identify the highly predictive risk 
factors for students in their own systems. 

The Fall River Study 

In a pioneering research project conducted during the early 1990s, Roderick studied the 
educational career pathways of a cohort of begirming 4th grade students in Fall River, a 
small urban school district in Southeastern Massachusetts. She examined academic 
performance, school engagement, and social background factors both for students who 
eventually dropped out and for students who eventually graduated. Roderick made 
several important discoveries: 

First, she discovered comparing all dropouts to all graduates masks important variations 
among subgroups. At first, her analysis seemed to indicate that dropouts were 
immediately distinguishable by warning signs as early as the 4th grade: On average, 4th 
graders who would eventually drop out had substantially lower grades compared with 
their peers who would graduate. However, when she probed more deeply, Roderick 
discovered that the gap was being driven by two subgroups: 
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• The average 4th grade performanee of dropouts was being pulled down by a 
subgroup of very low performers — about one-third of all dropouts — who would 
eventually drop out before even making it to 10th grade. 

• The average 4th grade performanee of graduates was being pulled up by a 
subgroup of high performers who would graduate in the top two-thirds of their 
elass. 

Seeond, her analysis revealed that the distriet had two very distinet types of dropouts — 
those who left sehool between 7th and 9th grades (early dropouts), and those who left 
during 10th through 12th grades (later dropouts). Not only did members of these two 
groups leave sehool at different points in the pipeline, they also had very different 
edueational eareers — exhibiting different risk faetors at different points in time: 

• Early dropouts started out with mueh lower grades in elementary sehool. 

• However, later dropouts — who made up 65 pereent of all Fall River dropouts — 
had 4th grade attendanee and grades that looked no different from students who 
would eventually graduate in the bottom third of the elass. 

Third, Roderiek uneovered powerful evidenee that the transition years — during whieh 
students move from elementary to middle sehool and from middle to high sehool — had 
a deeisive impaet on students who would later drop out. During the transition to middle 
sehool, aeademie performanee deelined for nearly all students. But the deterioration was 
mueh steeper for students who eventually dropped out. The same thing happened later 
during the transition to high sehool. 

In effeet, the two transitions were like hurdles that some students tripped over, eausing 
them to fall behind their peers in the trek to graduation. Even though their grades did not 
deeline more than graduates’ during the non-transition years, most dropouts never made 
up the ground they lost during 6th and 9th grades. 

Roderiek found that edueational engagement diverged in middle sehool as well, but in a 
somewhat different pattern: Eater dropouts experieneed a dip in attendanee during 6th 
grade that eontinued to aeeelerate until the end of middle sehool and into high sehool. 

The Chicago Study 

The Consortium on Chieago Sehool Researeh eonduets revealing analyses using a 
longitudinal database that traeks every student who moves into and through the Chieago 
Publie Sehools (CPS). That researeh has enabled Consortium researehers to eonstruet 
new and valuable indieators for predieting whieh students will drop out of high sehool. 

For example, by eombining two of the four high sehool risk faetors they had observed to 
be strongly eorrelated with graduation (attendanee, GPA, eredits earned, and failing 
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grades), Consortium members have developed an on-traek indieator that signals when 9th 
graders are falling seriously off the traek to earning a diploma. A student is eonsidered 
on-traek at the end of 9th grade if he or she has aeeumulated enough eourse eredits to 
earn promotion to 10th grade while reeeiving no more than one F (based on semester 
marks) in eore aeademie subjeets.^® 

Further analysis has revealed the on-traek indieator to be an espeeially good predietor of 
whether students will graduate. For example, among students entering 9th grade in 1999, 
the four-year graduation rate for on-traek freshmen was 8 1 pereent, eompared with only 
22 pereent for those who fell off traek as 9th graders. Another way to look at it: The on- 
traek indieator was 85 pereent sueeessful in predieting whieh members of the freshmen 
elass would not graduate. 

The on-traek indieator has revealed other important lessons about how students move 
through the CPS pipeline. Last year the Consortium issued a report showing that, not 
surprisingly, students who had high test seores in 8th grade were more likely to stay on 
traek when they got to 9th grade. However, high 8th grade test seores were no guarantee 
that a student would weather the diffieult transition to high sehool, and viee versa: 

In the 2003-04 freshman elass, for example, of the students who entered with 
very high 8th grade test seores (those in the top quarter of their elass), almost one- 
quarter were off traek by the end of their freshman year. On the other hand, of the 
students who entered high sehool in 2003-04 with very low 8th grade test seores 
(those in the bottom quarter of their elass), more than 40 pereent were on traek by 
the end of freshman year.^^ 

Why don’t 8th grade test seores tell the whole story about who will fall off traek during 
9th grade? One reason is that high sehools themselves play a substantial role in 
determining whether students stay on traek, independent of the risk faetors students bring 
with them when they enter. The Consortium demonstrated that even after eontrolling for 
student-level risk faetors related to raee, gender, prior aeademie aehievement, family 
soeioeeonomie status and being overage for the grade, high sehools varied enormously in 
both their 9th grade on-traek rates and in their graduation rates.^° 

Moreover, in a separate study, Roderiek and Erie Cambum found that Chieago high 
sehools also vary widely on another important measure — rates of recovery from 9th 
grade failure — even after eontrolling for a wide array of student risk faetors.^^ 

The Philadelphia Studies 

Researehers also have begun to study the pathway to dropping out of Philadelphia’s 
publie high sehools. One series of studies by University of Pennsylvania researeher Ruth 
Curran Neild and eolleagues has shed important light on the role of 9th grade as a eritieal 
year on the road to graduation. 
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First, success or failure in 9th grade greatly impaets a student’s ehanees of graduating, 
even taking into aeeount a wide array of 8th grade risk faetors related to soeial 
baekground, previous aeademie performanee and sehool engagement.^^ In other words, 
9th grade was a “make-or-break” year for many dropouts — a finding that led the 
researehers to rejeet the notion that the timing of events related to dropping out is 
random. The researehers eoneluded, “Our ability to prediet dropout . . . inereases 
eonsiderably when we know how students fare during their high sehool transition year.”^^ 

Seeond, 9th grade aets like a kind of quieksand for many Philadelphia students. More 
Philadelphia dropouts leave during 9th grade than in any other grade level. That does not 
mean dropouts spend only one year in high sehool, however. In faet, fewer than 5 pereent 
of dropouts leave during their first year in high sehool. Beeause of very high 9th grade 
retention rates, the most eommon number of years dropouts spend in Philadelphia high 
sehools is three.^"^ 

Finally, eertain risk faetors that students bring with them into high sehool — being 
overage, having reading and math seores signifieantly below grade level, low 8th grade 
attendanee, and having failed eourses in middle sehool — inerease the odds that students 
will fail 9th grade. However, low attendance during the first 30 days of 9 th grade is a 
more powerful predictor than any 8th grade factor, including test scores, age and 
academic failure.^^ 

A eo-author of one those 9th grade studies, Robert Balfanz of Johns Hopkins University, 
is eurrently working with Liza Herzog, a researeher at the Philadelphia Edueation Fund, 
to investigate the next obvious set of questions related to predieting dropouts in 
Philadelphia: How early in middle school eould the distriet identify students at high risk 
for falling off the traek to graduation? What middle school risk faetors best prediet 
dropping out of high sehool? 

To answer those questions, the researehers built a longitudinal database to follow a eohort 
of students who entered 6th grade in 1996-97, traeking their progress up until 2003-04, 
one year after they should have graduated. 

First, Balfanz and Herzog eondueted an initial sereen of about 20 student eharaeteristies 
to identify high-yield risk faetors — ones for whieh 75 pereent of students with the 
eharaeteristie do not make it to 12th grade on time. Next, they examined the impaet of 
those high-yield risk faetors on outeomes at eritieal points along the pipeline — 8th grade 
test seores, age in 8th grade, 9th grade sueeess or failure, and graduation from high 
sehool in four years or five years. 
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Although that project is still incomplete, it already has produced several highly important 
findings: 

• Four 6th grade risk factors proved to be powerful predictors of falling off the 
track to graduation — low attendance (80 percent or lower), a failing mark for 
classroom behavior, a failing grade in math, and a failing grade in English; 

• Sixth graders with any one of those risk factors had only a 10 percent chance 
of graduating on time and only a 20 percent chance of graduating a year late; 
and 

• Such students also were far more likely to perform poorly on state 
assessments, become overage during middle school and fail 9th grade — 
“often for several years. 

The results seem to support what the Fall River study found: The transition to middle 
school is as critical as the transition to high school, and 6th grade risk factors can be 
powerful predictors of dropping out. In fact, Balfanz and Herzog found that they could 
predict nearly half of all Philadelphia dropouts as early as 6th grade. 
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Table 1. Examples of Risk Factors that Significantly Increased the Odds of 
Dropping Out of High School from District-Level Studies 



Type of Risk 
Factor 


Chicago 


Philadelphia 


Fall River 


Academic 

Performance 


• Receiving more 
than one grade 
of F in core 
academic 
courses or not 
earning enough 
credits to be 
promoted during 
9th grade 


• Earning an F in 
English or 
mathematics during 
6th grade 

• Failing one or more 
courses during 8th 
grade 

• Entering 9th grade 
with math or 
reading scores 
below 8th grade 
level 

• Being retained in 
9th grade 


• Significant drop in 
grade point average 
from 8th to 9th 
grade 

• Being retained in 
any grade during 
K-8 or in high 
school 


Educational 

Engagement 


(n/a) 


• Low attendance 
(80 percent or 
lower) during 6th 
grade 

• Receiving a failing 
classroom behavior 
mark during 6th 
grade* 

• Low attendance 
during 8th grade 

• Low attendance 
during the first 30 
days of 9th grade 


• Significant drop in 
attendance 
beginning in 6th 
grade and 
worsening in 
subsequent years 



* Recent longitudinal studies typically have not had good access to other measures of behavior, such as number of office referrals, that 
might prove predictive in addition to or in place of teacher reports. 
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Essential Lessons from Longitudinal Cohort Studies 

1. While there is no single pathway that every dropout follows, there are common 
patterns, common crisis spots in the pipeline and common signposts, too. 

The notion that dropouts can follow observable and predictable patterns on the journey to 
leaving school runs afoul of another bit of conventional wisdom. Just as many people still 
believe that dropping out is caused entirely by individual social factors, many also 
believe that every dropout follows a unique journey — in other words, that dropping out 
is such a highly idiosyncratic, personal and complex phenomenon that no common 
patterns could ever be identified. 

Roderick contends that her Falls River study “challenges the assumption that dropping 
out is largely an individualized phenomenon [with] ‘many different routes.’ . . . [If] the 
path to dropping out could be best characterized as one that is different for every youth 
... we would not have observed any clear patterns at all in trends in late-grade dropouts’ 
school performance.’’^^ And the more recent Chicago and Philadelphia studies clearly 
support her claim. 

At the same time, however, longitudinal studies also have revealed that — at least in the 
districts that have been studied so far — dropouts are not a monolithic group whose 
members all exhibit the same risk factors and follow the same pathway to dropping out. 

A district might find two or three distinct sub-groups among dropouts, each of which 
follows a distinct pathway, along with a much smaller subset whose members exhibit no 
risk factors prior to dropping out. 

2. Longitudinal studies consistently identify a certain set of general educational risk 
factors as highly predictive of dropping out. 

The Fall River, Chicago, and Philadelphia studies did not identify new and surprising 
kinds of educational risk factors, but rather confirmed the general findings of earlier 
studies: Students who experience low or declining levels of either academic performance 
or educational engagement are much more at risk of falling off track and dropping out of 
school. 

3. However, it would be risky to simply guess at the exact measures of academic 
performance and educational engagement that best predict dropping out for 
students in a particular grade level in a particular district. 

For example, Balfanz and Herzog found that, when it came to academics, classroom 
grades trumped standardized test scores in predicting which 6th graders would someday 
drop out of Philadelphia high schools. As for engagement, classroom behavior marks 
turned out to be a much better predictor than the number of suspensions a student 
received. 
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Of course, that does not mean that standardized test scores are never useful. For example, 
in a study of risk factors for failing 9th grade — the biggest single predictor of dropping 
out for Philadelphia students — Neild and Balfanz found that, “In comparison to students 
who score at the 6th grade level or below in mathematics, those who score at the 7th 
grade level have odds [of failing 9th grade] that are lower by almost a quarter; those at 
the 8th grade level have odds of non-promotion that are lower by 42 percent . . . and those 
at the 9th grade level or above have odds that are lower by almost 60 percent.”^^ 

Moreover, there are a few factors — such as low attendance, being overage for the grade 
and failing 9th grade — that show up as important predictors of dropping out in nearly 
every single study. 

4, There might not be one single “leading indicator” that all dropouts exhibit first. 

It would be easier to prediet dropping out if all dropouts exhibited a eommon risk faetor 
that started them on the path to leaving sehool. Indeed, several earlier studies suggested 
that dropping out is a proeess that begins when students fall behind aeademieally, after 
whieh they beeome frustrated and disengaged from sehool."^° 

However, Balfanz and Herzog found that Philadelphia dropouts ean begin by exhibiting 
problems either with aeademie performanee or with sehool engagement during early 
middle sehool. “We saw 6th graders with poor behavior and attendanee who were not 
failing math and English,” says Balfanz. “And we saw students who were failing, but 
who did not have low attendanee or behavior problems. That leads us to speeulate that at 
the beginning of middle sehool, there might be an aeademie track and a non-academic 
track to failure. However,” he adds, “one set of problems ean lead to the other, problems 
ean eompound, and the traeks ean merge later on.”^^ 

He points out, “That also means you ean’t just use straight aeademie predietors alone at 
the 6th grade level, beeause you will be missing a group of kids for whom demographies 
and adoleseenee are ereating problems that will lead to aeademie struggle later on.” 

In other words, for many dropouts, “one thing ean lead to another.” But it is not always 
the same thing. 

5. Dropouts differ not only in how they begin on the path to dropping out, but where 
they end it. 

As Roderick’s Fall River study found, dropouts in a district can “clump” into subgroups 
based on the grade level in whieh they leave sehool, and those subgroups can exhibit very 
different educational pathways to dropping out. For example, she found that Fall River 
dropouts who left before 10th grade exhibited clear warning signs as early as 4th grade, 
but later dropouts showed their clearest warning signs during the transitions to middle 
school and high school. 
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6. The transition to middle school and high school is a critically important time for 
students. Many dropouts begin to show warning signs and downward trends in 6th 
grade or 9th grade. 

Students face many changes when they transition from elementary to middle school and 
from middle school to high school. In addition to having to negotiate a new and often 
larger institutional setting, students find that coursework is more intellectually 
demanding, teachers are less supportive, peer groups are larger, relationships are more 
complicated, and temptations become greater at the same time that they begin to 
experience more personal freedom. 

Not surprisingly, many students, not just eventual dropouts, experience declines in 
academic performance or school engagement during the transition years. But students 
who will eventually drop out often experience especially big declines from which they 
never recover. 

7. Transition-year failure often begins very early, and powerful risk factors can be 
observed even during the first month of school. 

Neild and Balfanz found low attendance during the first 30 days of 9th grade to be a more 
powerful predictor of later failure for Philadelphia high school students than any risk 
factor they exhibited in 8th grade."^^ And Elaine Allensworth and John Easton note that 
the correlations between 9th grade risk factors and dropping out in Chicago are only 
slightly less predictive when measured at the end of the first semester as they are when 
measured at the end of the year.'^^ 

8. Students with poor prior achievement and behavior are more likely to fail during 
transition years. However, it can be hard to predict transition-year failure based 
solely on information collected the year before students change schools. 

In her Fall River study, Roderick found that “students for whom difficulty following 
school transitions was a defining event were not necessarily those who were doing the 
most poorly in school before the school move.”'^'^ For example, when they were 4th 
graders, later dropouts had grades and attendance comparable to graduates. However, the 
decline in their academic performance during 6th grade was almost three times as big as 
the decline experienced by their peers who went on to graduate. 

Similarly, Consortium researchers found that low 8th grade test scores do not 
automatically doom students to 9th grade failure, and vice versa: 
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Almost one-quarter of students who began high school with high 8th grade 
achievement (those in the top quarter of their class) were not on track at the end of 
their freshman year. Although these students graduated from [8th grade] with 
strong test scores, only 37 percent of these off-track students graduated from high 
school within four years."^^ 

9. However, that does not necessarily mean that educators should wait until 9th 
grade to begin looking for warning signs. 

Balfanz believes the results of the current Philadelphia study call for a secondary school 
response to the high school dropout problem. Among Philadelphia dropouts who 
exhibited warning signs prior to 10th grade, the largest group by far was the group whose 
members exhibited them in 6th grade.^^ 

Moreover, additional research conducted by Roderick in Fall River suggested that “better 
performance during the transition to middle school would substantially reduce the 
likelihood of dropping out.”^^ 

10. School factors have an independent influence on whether incoming freshmen 
will graduate from high school or fall off track on the way to a diploma. 

Researchers affdiated with the Consortium on Chicago School Research have 
demonstrated that it is possible to calculate “adjusted” versions of several important high 
school outcome measures — staying on track in 9th grade; being promoted to 10th grade; 
recovering from 9th grade academic difficulty; and four- and five-year graduation rates. 

Since high schools still vary widely even in their adjusted outcomes, districts can use 
such data to identify how many and which high schools have an especially negative 
impact on whether students stay on track to graduate, and vice versa. 
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Part IV. Putting It All Together: Advice and Key Policy 
Questions for Building an Early Warning Data System 



The previous two seetions provided an overview of the lessons that long experienee and 
reeent researeh have taught us about the feasibility of identifying potential dropouts. This 
seetion addresses how polieymakers ean put those lessons into aetion. 

We now know that research is as important as development for building a good early 
warning data system. Skipping the researeh step ean severely undermine development of 
an effeetive and eost-effieient system to identify dropouts for intervention. As the authors 
of one of the Philadelphia studies put it, “We argue that longitudinal studies of dropout 
are enormously useful in identifying partieular points in students’ edueational eareers 
where sehool problems worsen or begin to manifest themselves.’’"^® 

Therefore, sehool systems should start by replieating the kind of researeh projeet 
eurrently being eondueted in Chieago and Philadelphia — before undertaking the 
development of a data system to provide early warnings to edueators and before 
implementing intervention strategies. 

Of eourse, that does not mean starting from serateh or working entirely alone. As 
deseribed below, the existing researeh base provides a very solid foundation to build on. 
And there is no reason to suspeet that groups of geographieally and demographieally 
similar distriets eannot eollaborate on researeh and development, share findings, and 
borrow from one another. 

In general, however, building an effeetive and effieient early warning data system will 
require a two-phase process: 

• Phase I — Researeh: Condueting a eohort-based, longitudinal study aimed at 
identifying the most effeetive and effieient risk faetors for a given sehool 
system, uneovering patterns in how those risk faetors play out for students 
over time, and assessing how to measure the impaet of sehools. 

• Phase 2 — Development: Leveraging the knowledge gained in Phase I to 
ereate an eleetronie data system to inform intervention efforts. 

That might sound as if doing this right will require a sizeable up-front investment in time 
and money. However, as we describe below, the research phase can take a very short 
amount of time and require relatively few resources. 
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Phase I. Conducting a Longitudinal Cohort Study 



Many education officials have the mistaken impression that conducting a longitudinal 
cohort study will require first developing a sophisticated student tracking system, after 
which they will have to wait for six or seven years as the system follows a cohort of 
students through the pipeline. In other words, it would take a lot of up-front time and 
money before you could begin to learn anything. 

Fortunately, that’s not the case. The methods used by researchers conducting longitudinal 
analyses in places like Fall River and Philadelphia reveal a much quicker, cheaper 
alternative. Those researchers collected existing information on previous cohorts whose 
students had already moved through the school system by engaging in a kind of “paper 
chase” — pulling data from the student records that every school district maintains in 
paper files or electronic databases. 

That means districts can save time and money by examining data on past cohorts to 
identify good predictors of what will happen to students in future cohorts. For example, a 
system that wanted to begin building a cohort data set to examine next fall would obtain 
information on the group of students who began as 6th graders during 1998-99. Those 
students should have graduated from high school in 2004-05. Data from 2005-06 will 
provide information on any of them who took an additional year to graduate. 

Step 1: Collect the Necessary Information 

Where Do We Obtain the Information? 

Data for conducting the Phase I analysis will be much easier to obtain in districts that 
have been using a centralized, electronic database for maintaining comprehensive records 
on individual students since the mid-1990s. However, while some districts began moving 
from paper records to electronic records a long time ago, others have yet to take the first 
step. 

In those districts, the information necessary to conduct the Phase I study will be spread 
across paper files maintained by multiple central office departments. In that case, analysts 
will have to spend time “going from door to door” to obtain the data. However, the 
information-gathering process need not take forever even under those circumstances. 
Balfanz reports that it took only about a month to accomplish it in Philadelphia, even 
though they had to obtain the information from multiple files: 

• A basic demographic file with race, gender and special ed status; 

• An enrollment file with information on whether, at the end of the prior year, 
students were still enrolled in school, had actively withdrawn or had 
graduated; 
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• A transcript file with information on classroom grades; 

• An attendance file; and 

• A test score file. 

Which Starting and Ending Points in the Pipeline Should We Look At? 

Analysts could theoretically pick any beginning grade level to define a eohort. Indeed, 
several studies have linked student aehievement and behavior in 1st grade to dropping out 
during high sehool.'^^ However, the reeent longitudinal studies deseribed above suggest 
that the best starting point is probably the transition into middle sehool. In most distriets, 
that will be 6th grade. 

Note that Roderiek found that deelines in grades and attendanee from late elementary 
sehool to middle sehool also revealed important information. That need not mean 
beginning with a full eohort of 5th graders. Distriets ean eolleet a subset of previous-year 
aeademie, attendanee, and behavior data for their eohort of 6th graders to answer 
questions about “how they were doing” before they got to middle sehool. 

Defining the end point might seem obvious. However, eonsider the Philadelphia studies 
suggesting that in some big eity sehool distriets, students spend more than one year, and 
sometimes up to three years, in 9th grade. Moreover, a federal eohort study traeking a 
national sample of 8th graders found that dropping out is seldom as simple as leaving and 
never returning. A large number of dropouts reenter the system, sometimes drifting in 
and out several times. 

Therefore, analysts should plaee the ending point for eolleeting data on a eohort at least a 
year after the students in that eohort should have graduated, if only to look for differenees 
in risk faetors based on four-year (on-time) and five-year graduation rates. Of eourse, 
they might not find big differenees, and that would be important to know as well: The 
Consortium on Chieago Sehool Researeh found that its 9th grade on-traek indieator 
worked just as well at predieting four-year graduation rates as five-year graduation 
rates.^^ 

This is not an argument for what kinds of dropout rate definitions should be used for 
publie reporting and aeeountability, whieh is beyond the seope of this white paper. 

Indeed, polieymakers might find it advisable to use a striet, on-time, four-year graduation 
rate for sueh purposes. The overriding goal of the Phase I study is simply to learn as 
mueh about a distrief s dropouts as possible. 



Achieve, Inc. 



June 2006 



Page 21 




How Many Cohorts Should We Examine? 

Systems will have to analyze data on at least one additional eohort to verify the findings 
of the Phase I study.^^ This is espeeially important to keep in mind when planning 
information eolleetion in distriets where data must be obtained from paper reeords 
maintained by different departments. 

What Pieces of Information Should We Include? How Many Possible Risk Factors 
Should We Test? 

As discussed above, the current Philadelphia study suggests that it’s advisable not to 
make untested assumptions about possible education-related risk factors before 
conducting the Phase I study. For example, one might assume that, among measures of 
academic performance, standardized test scores would reveal as much or even more 
about risk for dropping out as classroom grades. Conveniently, standardized test scores 
are often easier to obtain since they are more often centrally maintained. 

However, Balfanz and Herzog found just the opposite: Classroom grades are much better 
at predicting which Philadelphia 6th graders will drop out than are standardized test 
scores. The vast majority of Philadelphia students perform poorly on standardized tests, 
Balfanz reasons, but it is hard to actually fail 6th grade since teachers in middle schools 
are still fairly forgiving and often pass students based on effort. Therefore, classroom 
grades might be a better discriminator for identifying students who are truly falling off 
the track to earn a diploma. 

Similarly, when it came time to measure the predictive power of factors related to 
educational engagement, Balfanz and Herzog found that marks given by teachers for 
classroom behavior were much more predictive than the number of suspensions a student 
received. Behavior marks are cumulative, reflecting student attitudes and actions over the 
course of a semester or year, while suspensions reflect sporadic occurrences. Also, 
behavior marks reflect what is taking place inside the classroom — whether students are 
paying attention and are engaged in learning — while suspensions are often given for 
serious infractions that happen outside the classroom (before the first bell, between 
classes or after the last bell at the end of the day).^'^ 

Such surprising, counterintuitive findings reveal why it is important to begin the analysis 
by examining a large pool of potential indicators. For example, Balfanz and Herzog 
conducted a preliminary screen of about 20 student characteristics to obtain a final list of 
only four high-yield risk factors. 
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Table 2 provides a list of high-priority data elements to obtain for a Phase 1 analysis in 
eaeh of the three eategories of individual risk faetors — soeial baekground, aeademie 
performanee and edueational engagement. Note that this is a “birds-eye view” of possible 
data elements. While we have attempted to provide examples of speeifie data to look for 
under broad eategories like “diseipline” — sueh as elassroom behavior marks or offiee 
referrals — information maintained on students ean vary widely from distriet to distriet. 

A distriet that does not have any of the examples listed might still find alternative pieees 
of information that measure the same thing. 
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Table 2. 


Top-Priority Student-] 


i^evel Data Elements To Obtain for Phase I Analysis 




Social Background 


Academic Performance 


Educational Engagement 


Middle 

School 


1. SES — FRPL 
eligibility, family income, 
etc. 

2. Race/ethnicity 

3. Gender 

4. Mobility — number of 
schools enrolled 

5 . Y ears overage for grade 


1. Grades in academic subjects 
including at least English and 
math by end of quarter, semester 
and year 

2. Failing grades in math and 
English 

3. Scores on standardized 
assessments in at least reading 
and math, including grade-level 
and benchmark assessments 

4. Number of times retained in 
grade during elementary and 
middle school 


1 . Attendance — number of days 
or percentage of days absent 

2. Discipline problems — 
indicators of poor behavior, 
including, for example: 

a) classroom behavior marks 

b) number of office referrals 

c) number of counseling 

referrals 

d) number of suspensions 

3. Grades in non-academic subjects 
such as art, music and phys ed, 
aggregated into one score 
(Roderick, 1993) 


High 

School 


1. SES — FRPL 
eligibility, family income, 
etc. 

2. Race/ethnicity 

3. Gender 

4. Mobility — number of 
schools enrolled 

5 . Y ears overage for grade 


1 . Grades in core academic 
subjects, by end of quarter, 
semester and year 

2. Number of courses failed and 
passed in core courses, by end of 
quarter, semester and year 

3 . Number of credits attempted 
by semester, by year and 
cumulatively 

4. Number of credits earned by 
semester, by year and 
cumulatively 

5. GPA by semester, by year and 
cumulatively 

6. Ninth grade “on-track 
indicator” equivalent to or 
adapted from measure developed 
by Consortium on Chicago 
School: 

a) earned enough credits to be 
promoted and 

b) received not more than one 
semester F in core academic 
subject, with same calculated for 
subsequent grade levels 

7. On-time promotion to 10th 
grade 

8. Scores on standardized 
assessments, including grade- 
level, end-of-course, benchmark 
assessments and exit exams 

9. Dropped out previously and re- 
enrolled 


1 . Attendance — number of days 
or percentage of days absent 

2. Discipline problems — 
indicators of poor behavior, 
including, for example: 

a) number of office referrals 

b) number of counseling referrals 

c) number of suspensions 



Note: The numbering of data elements in this table is not meant to imply rank ordering on the basis of priority. All elements in this 
table are high priority. The relative predictive power of any element will vary by location. 
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Step 2: Analyze the Information To Identify High-Yield Risk Factors 

The goal of this step is to identify a group of high-yield risk faetors, ones that do the best 
job — and a good enough job — predieting whieh students will drop out. In their 
Philadelphia study, Balfanz and Herzog deeided to define “high yield” as a 75 pereent or 
greater ehanee of not graduating from high sehool on time (i.e., 75 pereent or more of the 
students who had the risk faetor did not graduate on time). 

The analysis at this point is not eomplieated. Analysts simply follow students’ time, 
examining outeomes for subgroups within the eohort that exhibit various eharaeteristies, 
determining what proportion drop out or graduate, as well as what proportion of students 
who do not exhibit a given risk faetor drops out or graduates. 

However, if that method does not identify high-yield risk faetors, analysts ean then begin 
to look for them in more eomplex eonfigurations of data (and, to hone the analysis, they 
might want to examine more eomplex eonfigurations anyway): 

a) Combinations of Indieators . For example, analysts ean examine what happens to 
students who exhibit different combinations of multiple risk faetors. At this point, 
analysts should examine not just whether edueational eharaeteristies (aeademie 
performanee and edueational engagement) are higher yield in eombination, but 
also whether edueational risk faetors are more predietive for males than for 
females or for some raeial/ethnie groups more than others. For example, they 
might examine eombinations, sueh as the following: 

• Failing grade plus low attendanee 

• Failing grade plus male 

• Failing grade plus low attendanee, male and Hispanie 

b) Changes in Indieators . Sinee several studies have found steep deelines in 
aeademie performanee or edueational engagement to be good predietors of falling 
off eourse for graduation, partieularly during transition years, analysts might want 
to examine ehanges in student eharaeteristies from one year to the next. For 
example: 

• Differenee in GPA (or some kind of mathematieal average refleeting overall 
aeademie performanee) between 5th and 6th grades 

• Differenee in attendanee between 5th and 6th grades 

Note that the bulleted examples above are just that — examples of possible 
eonfigurations of data to examine. In other words, there is not one formulaie set of steps 
for examining student eharaeteristies to identify high-yield risk faetors. Rather, analysts 
must take a more exploratory approaeh. However, as Balfanz points out, “If you have 
done a good job eompiling a robust set of soeial baekground data, aeademie performanee 
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data, and behavioral data, you can create any variations or combinations you might want 
to explore.”^^ 

Step 3. Conduct a Pipeline Analysis Based on the High-Yield Risk Factors Identified in 
Step 2 

Analysts should observe the pathways that students with high-yield early warning signs 
follow as they progress through the pipeline. Are there any especially troublesome grade 
levels — such as the transition years — where indicators of academic performance or 
educational engagement begin to plummet? What is the impact of high-yield risk factors 
in determining intermediate outcomes along the pipeline before students drop out, such as 
staying on track or failing the 9th grade? 

For example, after Balfanz and Herzog winnowed their list of potential indicators down 
to four “high-yield” risk factors, they next examined the impact of those four risk factors 
on key outcomes throughout the rest of the pipeline, including 8th grade test scores, ages 
at 8th and 10th grade promotion, and on-time and five-year graduation. 

Analysts also should look for patterns in the grade level and year that students drop out of 
high school. Are there distinct subgroups of early leaving and late leaving dropouts? How 
large is each group? How do their educational pathways and warning signs differ? 

Note that Philadelphia studies found that the year in which students left high school 
might not be obvious from looking at the grade in which they left high school. Therefore, 
examining both together might yield additional knowledge about a school system’s 
dropouts. 

The goal of this stage is to paint a vivid portrait of what the pathways to dropping out 
look like in a given school district. That analysis will help them more clearly understand 
how early risk factors work over time, as well as helping them make recommendations to 
district leaders for how the early warning data system should function and at which grade 
levels the school system should provide intensive interventions. 

Step 4. Assess How Effective and Efficient Highest-Yield Indicators Are, as well as the 
Potential Payoff from Interventions Triggered by Them 

Recall that any given risk factor or combination of risk factors only measures the 
likelihood that a given student will drop out. In other words, a risk factor is never a true 
prediction, but rather an assessment of how much risk an individual student faces based 
on his or her membership in a group whose educational outcomes we can predict much 
more accurately based on past cohort data. 

Therefore, policymakers and system leaders can use hypothetical scenarios to assess what 
various high-yield risk factors might produce if used as predictors to trigger 
interventions. 
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This process begins with knowing the answers to several questions about a risk factor: 
What percentage of students with the risk factor dropped out? What percentage 
graduated? What percentage of students without the risk factor dropped out? What 
percentage graduated? Using that information, system leaders can assess what proportion 
of dropouts would have been correctly targeted for interventions if a given risk factor or 
combination of risk factors had been used as the trigger. 

This is especially important for planning purposes because it helps leaders assess both the 
potential effeetiveness and potential effieieney of various risk faetors: 

The lower the number of “false negatives” (students a risk factor did not identify but who 
did drop out), the more effective the risk faetor is. Zero false negatives means a distriet 
would have identified and provided interventions to all students who eventually would 
have beeome dropouts. While it’s not aetually possible to get the number of false 
negatives down to zero in practiee, the higher the dropout rate for students that have a 
risk faetor, the more effeetive the risk faetor. 

Similarly, the lower the number of “false positives” (students a risk faetor would have 
identified but who did not drop out), the more efficient the risk faetor is. Zero false 
positives means a distriet would not have provided interventions to any students who 
would have graduated anyway, thus saving dollars.^^ 

Figure 1, How Risk Factors Identify False Negatives and False Positives 
False negatives False positives 
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Dropouts 
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Identified by risk factor 



Consider the following hypothetieal example based on a real risk faetor — the 9th grade 
on-traek indieator ereated by the Consortium on Chieago Sehool Researeh. The 
Consortium reports that 8 1 pereent of students who were on traek by the end of 9th grade 
eompleted high school in four years, eompared with only 22 pereent of 9th graders who 
finished off traek. Based on that information, it’s possible to ealculate that the on-traek 
indieator eorreetly predieted 85 pereent of eventual dropouts among that group of 
entering freshmen. 

Now assume that the CPS deeided to provide targeted assistanee to all off-traek students 
at the end of 9th grade. Further assume that 30,000 freshmen enter Chieago high sehools 
eaeh year and about 58 pereent of them fall off track, which means the program would 
provide interventions to 17,370 students. The risk faetor would fail to identify 2,400 false 
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negatives, students who, despite being on traek at the end of 9th grade, would eventually 
fail to graduate. Those students would not reeeive the help they need. 

Conversely, the indieator would identify 3,821 false positives, off-traek students who 
would have reeovered and graduated anyway. Assume that the intervention program eosts 
an average of $350 per student. That would make the total eost of the program 
$6,079,500, of whieh $1,337,490 (or 22 pereent) would have been spent on false 
positives, students who didn’t need the extra help beeause they would have graduated 
even without it. 

However, if the program is even halfway effeetive (i.e., if it aehieves a 50 pereent sueeess 
rate for getting off-traek students baek on traek to graduate), the CPS would: 

• Reduee the number of dropouts from 15,948 to 9,174 (a 42 pereent reduetion); 

• Inerease the number of graduates from 14,052 to 20,826 (a 48 pereent 
inerease); 

• Reduee the distriet’s dropout rate from 53 pereent to 3 1 pereent; and 

• Inerease the distriet’s graduation rate from 47 pereent to 69 pereent (i.e., to 
about the national average).^^ 

While it is not perfeet, the on-traek indieator elearly is a potentially very powerful tool for 
eurbing dropout rates. And it is not the only powerful risk faetor at the distriet’s disposal. 

Sueh evaluations of the effeetiveness and effieieney of different risk faetors ean be 
eondueted with very few pieees of information — the dropout rate for students who have 
the risk faetor, the dropout rate for those who do not, the proportion of students who 
exhibit the risk faetor and the number of entering freshmen (or 6th graders). 

Step 5. Conduct a School-Level Analysis To Better Understand Where Individual Risk 
Factors Are Most Concentrated and Which Schools Put Students at an Even Greater 
Risk of Dropping Out 

Onee the most powerful student-level predietors of dropping out are known for a 
partieular sehool system, the distriet ean eonduet a sehool-level analysis. For example: 

a) Coneentration of risk faetors : Analysts should determine whieh middle sehools 
and high sehools reeeive espeeially high numbers of students with partieular high- 
yield risk faetors. 

b) Contribution to risk faetors : Analysts should identify middle and high sehools 
where transition years are most problematie. For example, they ean identify 
sehools where: 
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• High numbers of students exhibit deelines in aeademie performanee or 
edueational engagement during transition years; 

• High numbers of students who previously showed no warning signs 
exhibit sueh deelines during transition years; and 

• Students in general experienee espeeially steep deelines. 

e) Impaet on outeomes : Analysts should evaluate how mueh individual high sehools 
themselves eontribute to the dropout problem independent of student-level risk 
faetors. They ean do that by using regression analyses to “eontrol for” the risk 
faetors that students earry with them when they enter 9th grade to determine 
whieh high sehools have higher- than-predieted or lower- than-predieted outeomes. 
They also ean ealeulate adjusted rates on important outeomes for eaeh individual 
high sehool: 

• Adjusted rate of freshmen staying on traek (the on-traek indieator); 

• Adjusted 10th grade promotion rate; 

• Adjusted rate of reeovery among students who fall off traek in 9th grade; 
and 

• Adjusted four- and five-year graduation rates. 
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Phase II. Developing an Early Warning Data System 



What Kind of Database Will We Have To Buy or Build To Do This? 

To keep track of which students develop risk factors, when they develop them, and how 
they fare throughout the pipeline, school systems will have to build electronic record- 
keeping systems, or data warehouses, that maintain information on individual students 
and schools over time. Many states and districts have already begun to invest in the 
development of such systems, especially to meet the reporting requirements of No Child 
Left Behind. Ideally, the early warning data system should piggyback on such efforts. 

In general, such databases should include: 

• A unique student identifier that allows an individual student to be tracked 
by grade level and from elementary to middle to high school; 

• Accurate enrollment information on each student, including entry and exit 
by school attended; 

• Student demographic information, including eligibility for the federal free 
and reduced-price lunch program, race/ethnicity, gender, and age; 

• Student transcript information, including courses attempted, courses 
completed, grades, credits earned, and any instances of being retained in a 
grade; 

• Student attendance; 

• Student behavior grades or discipline records; and 

• Student graduation and dropout information. 

Districts will be in a much better position to construct such a database in states that have 
created statewide longitudinal data systems. For example, states can do a better job 
tracking whether a student who left a district dropped out entirely or enrolled in another 
district elsewhere. A publication entitled Creating a Longitudinal Data System, available 
online at www.achieve.org. provides greater detail on the necessary and desirable 
features of statewide longitudinal databases. 

Not surprisingly, the number of commercial database software programs on the market 
and the number of vendors offering customized database assistance have increased 
greatly over the last few years. District leaders who would like a general overview of 
such software, as well reviews of more than a dozen of the more popular commercial 
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packages, can consult the Data Use Web site maintained by Jeff Wayman of the 
University of Texas at Austin, http ://edadmin. edb.utexas.edu/datause/index.htm . 

Which Warning Signs Should the System Monitor and Report On? 

The answer to this question should be determined primarily by the results of the Phase I 
analysis. System leaders should be wary of taking shortcuts. The primary value of an 
early warning data system rests in its level of its predictive power. For example, analysts 
might find that grades are more highly predictive than test scores, but be told by 
department heads and budget officers that “test scores are just so much easier and cheaper 
to include.” 

Policymakers should examine the long-run trade-offs associated with following that 
advice. It might turn out to be much more cost-effective over the long run to include 
indicators that are more expensive to obtain and maintain in any given year but do a 
better job predicting which students are more likely to drop out. 

Furthermore, it might seem odd to include risk factors that rely on human judgment and 
that psychometricians, for example, would consider less “statistically reliable” because 
they are “more subjective” and “less standardized.” However, it is important to keep in 
mind that the primary purpose of an early warning data system is prediction and 
prevention, not measurement per se. Thus, the value of any piece of data lies not in its 
general “reliability” but in its specific validity — it’s predictive validity. 

Which Grade Levels Should the System Monitor and Report On? 

The answer to this question should be driven by two factors — the grade levels identified 
by the Phase I study as the most useful for predicting dropouts and the grade levels at 
which it is feasible to provide interventions. In general, systems should target 
intervention to the grade levels that are the weakest points in their secondary pipeline, the 
“crisis spots” that set students on the road to dropping out. 

Many districts are providing extra assistance to 7th and 8th graders to help them improve 
their standardized test scores. There can be little doubt that this is a positive development. 
Research by the Council of the Great City Schools and the National Center for 
Educational Accountability has found that monitoring student performance and providing 
extra academic assistance — as early and often as possible — can be an effective strategy 
for raising student achievement and closing achievement gaps between groups. 

Moreover, given that low academic performance generally puts students at greater risk of 
dropping out, such efforts might help reduce dropout rates as well. 

However, policymakers should not assume that such programs automatically constitute 
the best possible approach to reducing dropout rates or that they can necessarily “kill both 
birds with the same stone.” For example, Roderick found that, in Fall River, “better 
performance in non-transition years would have a small impact on the conditional 
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likelihood of dropping out. [But] better performanee during the transition to middle 
sehool would substantially reduee the likelihood of dropping out.” And Balfanz and 
Herzog found that far more Philadelphia dropouts first showed warning signs in 6th grade 
than in 8th grade, and that elassroom grades were better predietors of whether 6th graders 
would eventually drop out than were standardized test seores. 

What If a District Chooses Not To Undertake the Phase I Analysis? 

As diseussed above, analyzing whieh preeise risk faetors are most predietive of dropping 
out is relatively quiek, easy, and — best of all — inexpensive. And the long-term 
payoffs, both in terms of eeonomie effieieney and graduation outeomes — ean be huge. 
However, distriets that eannot undertake sueh an analysis for some reason should still 
eolleet information to analyze student- and sehool-level needs before spending signifieant 
resourees on programs meant to improve graduation rates. The major faetors that put 
students at risk of dropping out — low aeademie performanee and edueational 
engagement — are well established, and a distriet ean hedge its bets by eolleeting 
multiple kinds of information to measure eaeh of those faetors (for example, elassroom 
grades and standardized test seores). In sueh ease. Table 2 on page 24 ean serve as a 
helpful summary to the basie kinds of researeh-based information to eolleet on students. 

How Frequently Should the System Monitor and Report on Risk Factors? 

If feasible, distriets should monitor and report on warning signs in every middle and high 
sehool grade level on an annual basis. But they should eonsider more frequent monitoring 
and reporting if eertain grade levels turn out to be partieularly important stieking points 
on the road to graduation. 

For example, longitudinal studies have revealed two important additional findings about 

58 

transition years: First, few students reeover from failure during transition years. 

Seeond, aeademie failure and disengagement oeeur very early during the transition years. 
Therefore, system leaders should eonsider eonfiguring the early warning data systems to 
make multiple reports during transition years — at the end of every quarter or at the end 
of every semester. 

However, polieymakers should keep in mind that longitudinal studies also have taught us 
important lessons about the dangers of making untested assumptions and generalizing 
based on data from other plaees. A distriet’ s Phase I analysis eould eoneeivably reveal 
different pathways than those identified in the Fall River, Chieago and Philadelphia 
studies. Indeed, some national studies have found that the ehanee of dropping out 
inereases from 9th through 12th grades — a quite different pattern than in Philadelphia, 
where three in four dropouts leave before ever reaehing 1 1th grade.^^ 

Therefore, the decision about when and how often to report and act on warning signs 
should be driven primarily by what a system finds when it conducts its own Phase I 
pipeline analysis. 
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What Kinds of Data Should We Report to Whom? 

An early warning data system ean report many kinds of information helpful for redueing 
dropout rates. First of all, of eourse, the system ean report whieh individual students 
exhibit risk faetors to sehool or program staff But reports of aggregate risk faetors by 
sehool ean be helpful to sehool and distriet leaders as they plan and budget for 
intervention programs. And reports of adjusted outeome measures by sehool ean be 
useful to sehool-level eommittees working on improvement plans and to distriet offieials 
eonsidering sehoolwide interventions and reforms. 

Readers should eonsult the eompanion JFF paper, Improving Outcomes for Struggling 
and Out-of-School Youth, for a full overview of researeh on the effeetiveness of various 
student intervention programs and whole-sehool reforms for redueing dropout rates, as 
well as reeommended state and loeal polieies for supporting sueh efforts. However, here 
are several examples that illustrate how data from the Early Warning System might be 
useful for making sueh deeisions: 

a) Intervention: Extra help for eertain groups of students who share a partieular 
risk faetor. If many students exhibit risk faetors related to low aeademie 
performanee, for example, polieymakers might deeide to provide aeeelerated 
instruetion to sueh students in the form of eateh-up eourses. Johns Hopkins 
University has developed sueh eourses as part of its Talent Development 
model for middle sehool and high sehool reform, and reeent researeh suggests 
sueh eourses are partieularly effeetive at helping students get baek on traek 
aeademioally and, along with other student support, ean inerease eredit 
aeeumulation, 10th grade promotion and graduation rates.^° Other seeondary 
sehool reform models have developed sueh eourses as well; for example, 
Ameriea’s Choiee offers eourses sueh as Ramp Up to Eiteraey and Ramp Up 
to Algebra. 

b) Intervention: Espeeially intensive or personalized help for individual students. 
Students who exhibit warning signs related to edueational engagement — for 
example, very low attendanee or very poor behavior — might require one-on- 
one eounseling.^^ Sehools ean target students for eounseling based on data 
from the Early Warning System. (However, if high proportions of students 
exhibit sueh warning signs in some sehools, the distriet might eonsider 
sehoolwide interventions instead or in addition to one-on-one eounseling. 
Moreover, some sehoolwide reforms make it mueh easier to ensure that adults 
ean provide individual support to students instead of simply expeeting often 
overworked eounselors to do so. See C below.) 

e) Prevention: Sehool-level reforms that ean reduee student risk faetors and 
dropping out. Providing interventions to students who develop risk faetors ean 
help improve graduation rates. But distriet leaders ean also intervene on a 
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schoolwide level to ereate eonditions that help prevent students from 
developing risk faetors in the first plaee, and that reduee the negative impaet 
of some sehools on graduation rates. For example, are there middle sehools 
and high sehools where transition years are espeeially diffieult for students? 
Do students who enter sueh sehools exhibit big deelines in aeademie 
performanee, edueational engagement or both? If aeademie performanee, 
sehool or distriet leaders might eonsider eurrieulum ehanges, professional 
development, or earving out more time for math and literaey. If edueational 
engagement, leaders might eonsider restrueturing those grade levels into small 
learning eommunities, instituting adult advoeate or mentor programs, 
restrueturing sehedules to allow teaehers more time to interaet in supportive 
ways with individual students, ete. 
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Table 3. Uses of Student- and School-Level Information Generated by an Early 
Warning Data System^ 





Why School Staff Members Need 
To Know 


Why District Leaders Need To Know 


1) Risk factors by 
individual student 


Targeting students for individual 
and group interventions (e.g., 
individual counseling, catch-up 
courses in math and literacy, etc.) 




2) Aggregate risk factors 
by schooi and type of 
school 


Planning intervention programs 


Planning and budgeting for intervention 
programs 


3) Sizes and rates of 
declines in academic 
performance and 
educational engagement 
during transition years 


School improvement planning 


Targeting schools for institution-level 
interventions and reforms (e.g., 9th 
grade academies, freshmen seminars, 
interdisciplinary team teaching, etc.) 


4) School-level outcomes, 
raw and adjusted for 
student-level risk factors 

• On track by HS 
grade 

• Off track 
recovery rates 

• Graduation 
rates 


School improvement planning 


Targeting schools for institution-level 
interventions and reforms (e.g., changes 
in school organization, scheduling, 
staffing, curriculum, etc.) 


5) Systemwide analyses 
of student 
characteristics, risk 
factors and outcomes 




Planning for districtwide strategies that 
go beyond student- and school-level 
interventions. For example, analyzing 
student characteristics to craft a range of 
programs to meet the diverse needs of a 
wide range of young people — from 
dropouts just a few credits shy of 
graduating to overage, under-credited 
students repeating 9th grade. 
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Data produced by the system can have uses beyond direct intervention. For example, 
system leaders should strongly consider investing the resources to provide middle grades 
educators with reports showing how their graduates fare in high school — how many 
stayed on track as they progressed through high school and how many dropped out. The 
Consortium on Chicago School Research has offered detailed information about 8th 
grade graduates to Chicago’s elementary schools (Chicago uses a K-8 structure) since 
1999. Student outcomes are broken out by various factors, including 8th grade test scores 
and passing rates in math and English. Finally — and perhaps most important — the 
reports employ compelling and easy to understand graphics to convey complicated 
information (see Figure 2). 

Such reports might help bring middle school educators on board as systems seek to 
upgrade the mission of middle schools to focus on preparing all students for success in 
rigorous high school coursework. For example, surveys by the Southern Regional 
Education Board have found that fewer than half of middle school teachers in the region 
see their primary mission as preparing students for high school, and only 12 percent see it 
as preparing all students with the academic knowledge and skills needed to be successful 
in college-preparatory classes during high school.^^ 
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Figure 2, Chart from Consortium on Chicago School Research Report, “How Do 
Barton Graduates Perform in CPS High Schools?”®^ 




On Track Graduated Off Track Dropped Out Left CPS 

Notes: One symbol equals approximately 3 students. See Table 1 on the next page for precise numbers. 

Status is determined at the beginning of the following year. i.e.. freshman status is determined from data taken in the beginning of sophomore year. 
On track students had enough credits to odvarKe to the next grade on time and recelsed no more than one F in a core course. 



Note that the interventions and other data uses diseussed above are not meant to represent 
an exhaustive list of solutions to the dropout problem, but only to provide several 
examples of interventions that ean benefit highly from the kind of data system diseussed 
in this paper. In large urban distriets where risk faetors are pervasive and more than half 
of the student population drop outs, for example, individual interventions and even 
aggressive institutional reforms might not be enough to adequately address the problem. 
Sueh systems might need to invest in large-seale, systemwide strategies as well, sueh as 
ereating multiple institutional and non- institutional pathways to obtaining a diploma and 
a portfolio of flexible seeond-ehanee options for students who already have dropped out. 
More information on sueh strategies ean be found in the eompanion JFF paper. Improving 
Outcomes for Struggling and Out-of-School Youth. 

Some Final Words of Advice 

Y Build eapaeity to do this over the short term and long term. Small and rural 
distriets with fewer resourees and smaller, or nonexistent, researeh departments 
should eonsider working eollaboratively to share the eost of researeh and 
development. Federally funded assistanee eenters, regional edueation serviee 
ageneies and institutions of higher edueation all ean help by providing 
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technical expertise and advice. (Even systems with midsize to large research 
departments might want to consider partnering with researchers at nearby 
colleges and universities who are interested in the dropout problem.) And states 
can help by providing resources and by investing in statewide data systems that 
offer an informational backbone that local efforts can build upon. 

Districts should incorporate fields into the database that allow them to keep 
track of which students receive which interventions. That will allow them to 
evaluate how well the interventions are working, for whom and where. 

Districts should keep in mind that high-yield risk factors might change over 
time. For example, the demographic characteristics of a district’s student 
population could change, or the school board might enact policies that cause 
the risk factors or the timing of the risk factors to shift. However, once the 
electronic data system is in place, repeating the Phase I analysis will not be 
difficult or time consuming. 

Districts should add more information about students as it becomes available. 
For example, some school systems are using standardized quarterly benchmark 
tests to monitor student performance, and many of them maintain the results 
electronically. Failing to add new information to databases and analyses could 
mean missing out on identifying even more powerful predictors of dropping 
out. 

And Some Future Actions To Consider 

■C School system leaders should consider analyzing other high school outcome 
variables in addition to promotion and graduation. For example, another goal of 
high school reform is to increase the proportion of graduates who complete a 
sequence of rigorous college prep courses and score at college-ready levels on 
state assessments. What factors predict whether students are more or less likely 
to do so? Can school systems create risk factors and on-track indicators related 
to college-and-work readiness, too?^"^ 

■C Finally, state leaders should consider going “beyond the basics” to build a more 
expansive longitudinal data system. Some states have built P-16 databases that 
track individual students through the entire educational pipeline and even out 
into the workforce. The potential payoffs to such investments are tremendous. 
See Creating a Fongitudinal Data System, available at www.achieve.org, for 
additional information. 
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Part V. Conclusion 



Because of the rapidly changing American economy and a new commitment on the part 
of state leaders to raise graduation standards, solving the dropout problem has become 
more important than ever before. 

At the same time, we know that dropout prevention efforts of the past have yielded poor 
results. Part of the problem is that such efforts did not provide at-risk students with 
effective interventions (see the companion JFF paper. Improving Outcomes for 
Struggling and Out-of-School Youth), but another reason is that they did not accurately 
target the right students to receive extra help in the first place. 

Knowing which students are at greatest risk for dropping out and which schools most 
exacerbate the problem is the first step to reducing dropout rates. Fortunately, today’s 
education leaders have better research and data than were available 20 years ago. 

If policymakers heed the most current research, avoid the mistakes of the past, and invest 
sufficient up-front “research and development” dollars, they can build data systems to 
identify a good many students on the path to dropping out early enough to make a 
difference. And district administrators can intervene in schools that contribute the most to 
the dropout problem, changing them from institutions that “push students out” into 
challenging and supportive environments that keep teenagers in school and on track for a 
diploma. 
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